Most state-of-the-art approaches for weather and climate modeling are based on physics-informed numerical models of the atmosphere. These approaches aim to model the non-linear dynamics and complex interactions between multiple variables, which are challenging to approximate. Additionally, many such numerical models are computationally intensive, especially when modeling the atmospheric phenomenon at a fine-grained spatial and temporal resolution. Recent data-driven approaches based on machine learning instead aim to directly solve a downstream forecasting or projection task by learning a data-driven functional mapping using deep neural networks. However, these networks are trained using curated and homogeneous climate datasets for specific spatiotemporal tasks, and thus lack the generality of numerical models. We develop and demonstrate ClimaX, a flexible and generalizable deep learning model for weather and climate science that can be trained using heterogeneous datasets spanning different variables, spatio-temporal coverage, and physical groundings. ClimaX extends the Transformer architecture with novel encoding and aggregation blocks that allow effective use of available compute while maintaining general utility. ClimaX is pre-trained with a self-supervised learning objective on climate datasets derived from CMIP6. The pre-trained ClimaX can then be fine-tuned to address a breadth of climate and weather tasks, including those that involve atmospheric variables and spatio-temporal scales unseen during pretraining. Compared to existing data-driven baselines, we show that this generality in ClimaX results in superior performance on benchmarks for weather forecasting and climate projections, even when pretrained at lower resolutions and compute budgets. The source code is available at https://github.com/microsoft/ClimaX.

Machine learning and deep learning methods have been widely explored in understanding the chaotic behavior of the atmosphere and furthering weather forecasting. There has been increasing interest from technology companies, government institutions, and meteorological agencies in building digital twins of the Earth. Recent approaches using transformers, physics-informed machine learning, and graph neural networks have demonstrated state-of-the-art performance on relatively narrow spatiotemporal scales and specific tasks. With the recent success of generative artificial intelligence (AI) using pre-trained transformers for language modeling and vision with prompt engineering and fine-tuning, we are now moving towards generalizable AI. In particular, we are witnessing the rise of AI foundation models that can perform competitively on multiple domain-specific downstream tasks. Despite this progress, we are still in the nascent stages of a generalizable AI model for global Earth system models, regional climate models, and mesoscale weather models. Here, we review current state-of-the-art AI approaches, primarily from transformer and operator learning literature in the context of meteorology. We provide our perspective on criteria for success towards a family of foundation models for nowcasting and forecasting weather and climate predictions. We also discuss how such models can perform competitively on downstream tasks such as downscaling (super-resolution), identifying conditions conducive to the occurrence of wildfires, and predicting consequential meteorological phenomena across various spatiotemporal scales such as hurricanes and atmospheric rivers. In particular, we examine current AI methodologies and contend they have matured enough to design and implement a weather foundation model.

Climate change and extreme weather events have made weather and climate modelling a challenging yet crucial real-world task. While current state-of-the-art approaches tend to employ numerical models conditioned on physical information collected from the atmosphere, the development of powerful deep learning models and the increasing availability of massive climate datasets have advanced the possibility of a data-driven, general-purpose foundation model for such modelling. In the new paper ClimaX: A Foundation Model for Weather and Climate, a team from Microsoft Autonomous Systems and Robotics Research, Microsoft Research AI4Science and the University of California at Los Angeles presents ClimaX, a general-purpose deep learning foundation model for weather and climate that can be efficiently adapted for various tasks related to the Earth's atmosphere. The team set out to train a generalizable foundation model capable of handling heterogeneous datasets of different variables and providing spatiotemporal coverage based on physical groundings. They built ClimaX on a vision transformer (ViT) backbone and introduced two main architectural changes -- variable tokenization and variable aggregation -- to improve its flexibility and generality.

Over the last decade, artificial intelligence (AI) and machine learning (ML) techniques have developed at an unprecedented pace, and it is now evident that many scientific disciplines can hugely benefit from these developments provided they explore more data centric methodologies. The science community is currently exploring how the new AI and machine learning techniques can be exploited to further enhance our Earth-system prediction capabilities and first results show exciting potential. However, the scope and speed of these AI/ML developments also generate challenges for weather and climate modelling centres such as ECMWF. These challenges regard the necessary knowledge that needs to be established, the software and hardware infrastructures that need to be developed and used, and the integration of machine learning and conventional tools across the entire prediction workflows, which are continuously evolving. It is fundamental that these challenges are addressed and that the weather and climate modelling community and ECMWF's Member and Co-operating States are enabled to make the best possible use of machine learning in the years to come.